Logic behind crawling an webpages like that of Screaming Frog? [on hold]

Posted by sree on Programmers See other posts from Programmers or by sree
Published on 2014-08-20T18:02:29Z Indexed on 2014/08/21 22:26 UTC
Read the original article Hit count: 229

Filed under:

logic

|

web-crawler

|

crawlers

I would like to know what is the parameters to be considered while developing a crawler like that of Screaming Frog.

Am looking forward for information on do's and dont's of webpage crawling. What are the problems the crawler may infuse on the webpages like loadtime (maybe?) or anything that effects webpage during crawling.

What are the rules the crawler needs to follow etc. Basically anything info that makes the crawler look good and accurate. Just point me in a right direction to achieve it..

Hope my requirement is clear this time.. :)

© Programmers or respective owner

Related posts about logic

Business Logic Layer Pattern on Rails? MVCL

as seen on Stack Overflow - Search for 'Stack Overflow'
That is a broad question, and I appreciate no short/dumb asnwers like: "Oh that is the model job, this quest is retarded (period)" PROBLEM Where I work at people created a system over 2 years for managing the manufacture process over demand in the most simplified still broad as possible, involving… >>> More
What is the correct way to deal with similar but independent features?

as seen on Programmers - Search for 'Programmers'
Let's say we have a feature request come in and we begin work on it, which we'll call feature-1. It introduces some new logic to the application, which we'll call logic-A and logic-B. A programmer branches from the release branch and begins work on the feature. Soon after, we get another feature… >>> More
What is up with the Joy of Clojure 2nd edition?

as seen on Programmers - Search for 'Programmers'
Manning just released the second edition of the beloved Joy of Clojure book, and while I share that love I get the feeling that many of the examples are already outdated. In particular, in the chapter on optimization the recommended type-hinting seems not to be allowed by the compiler. I don't know… >>> More
Transforming Business Logic into Web Services Using DB2 9.5 and IBM Data Studio

as seen on Internet.com - Search for 'Internet.com'
Learn how to take business logic such as stored procedures, functions, or SQL statements and turn them into a Web service. >>> More
Identifier for the “completed” stage of a process: 0, 99, something else?

as seen on Programmers - Search for 'Programmers'
Say, that you are handling a multi-step process (like a complex registration form, with a number of steps the user has go through in order). You need to be able to save the current state of the process (e.g. so the user can come back to that registration form later and continue form the step where… >>> More

Related posts about web-crawler

web crawler needed

as seen on Stack Overflow - Search for 'Stack Overflow'
does anybody know where i can get a free web crawler that actually works with minimal coding by me. ive googled it and can only find really old ones that dont work or openwebspider which doesnt seem to work. ideally id like to store just the web addresses and which links that page contains any suggestions… >>> More
Building an automatic web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
I am building a web application crawler that's meant not only to find all the links or pages in a web application, but also perform all the allowed actions in the app (such as pushing buttons, filling forms, notice changes in the DOM even if they did not trigger a request etc.) Basically, this is… >>> More
Appengine Apps Vs Google bot web crawler

as seen on Stack Overflow - Search for 'Stack Overflow'
i built an appengine web app cricket.hover.in. The web app consists of about 15k url's linked in it, But even after a long time of my launch, no pages are indexed on google. Any base link place on my root site hover.in are being indexed with in minutes. but i placed the same link home page of root… >>> More
Extracting data from internet

as seen on Programmers - Search for 'Programmers'
I would like to extract data from internet like www.mozenda.com does but I want to write my own program to do that. Specific data I'm looking for is various event data. Based on my research, I think custom web crawler is my answer but I Would like to confirm the answer and see if there are any suggestion… >>> More
Web crawler update strategy

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to crawl useful resource (like background picture .. ) from certain websites. It is not a hard job, especially with the help of some wonderful projects like scrapy. The problem here is I not only just want crawl this site ONE TIME. I also want to keep my crawl long running and crawl the updated… >>> More